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ABSTRACT 

High resolution calibrated near infrared (NIR) imagery of the Space Shuttle Orbiter was obtained during hypervelocity 
atmospheric re-entry of the STS-119, STS-125, STS-128, STS-131, STS-132, STS-133, and STS-134 missions. This 
data has provided information on the distribution of surface temperature and the state of the airflow over the windward 
surface of the Orbiter during descent. The thermal imagery complemented data collected with onboard surface 
thermocouple instrumentation. The spatially resolved global thermal measurements made during the Orbiter’ s 
hypersonic re-entry will provide critical flight data for reducing the uncertainty associated with present day ground-to- 
flight extrapolation techniques and current state-of-the-art empirical boundary-layer transition or turbulent heating 
prediction methods. Laminar and turbulent flight data is critical for the validation of physics-based, semi-empirical 
boundary-layer transition prediction methods as well as stimulating the validation of laminar numerical chemistry 
models and the development of turbulence models supporting NASA’s next-generation spacecraft. In this paper we 
provide details of the NIR imaging system used on both air and land-based imaging assets. The paper will discuss 
calibrations performed on the NIR imaging systems that permitted conversion of captured radiant intensity (counts) to 
temperature values. Image processing techniques are presented to analyze the NIR data for vignetting distortion, best 
resolution, and image sharpness. 

Keywords: HYTHIRM, Space Shuttle thermography, hypersonic imaging, near infrared imaging, histogram analysis, 
singular value decomposition, eigenvalue image sharpness 

1. INTRODUCTION 

The NASA Langley Research Center led HYpersonic THermodynamic InfraRed Measurement (HYTHIRM) project 
culminated in the acquisition of high-resolution calibrated infrared imagery of the Space Shuttle Orbiter during 
hypersonic atmospheric entry. The thermal imaging provided a unique and never before observed perspective on the 
global distribution of surface temperature and the state of the airflow (i.e., laminar/turbulent) over the entire windward 
surface of the Orbiter during re-entry. Laminar and turbulent flight data are critical for the validation of physics-based, 
semi-empirical boundary- layer transition prediction methods, as well as stimulating the validation of laminar numerical 
chemistry models and the development of turbulence models supporting NASA’s next-generation spacecraft [1]. This 
paper will provide details behind the use of a digital near infrared (NIR) imaging system and describe the 
characterizations and procedure performed to obtain quantitative temperature imagery. Past attempts by the Cast Glance 
aircraft to image the Orbiter during STS- 121 and STS-1 15 missions have resulted in successful image acquisitions, 
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Figure 1. STS- 121 and STS-1 15 images showing early promise of imaging during hypersonic re-entry. 


however, the quality of the acquired data allowed for limited data processing [2-5]. During both missions, the Orbiter 
imagery was obtained using an analog-based NIR imaging system and due to image saturation, limited quantitative 
analysis could be performed. The STS-121 and STS-1 15 acquired images are shown in Figure 1 with pixels at nose, 
leading edges, and tail indicating saturation. The early results, however, indicated significant potential for imaging the 
Space Shuttle during hypersonic re-entry. As a result, a digital NIR imaging system was implemented after a series of 
rigorous testing at the Solar Tower Thermal Test Facility at Sandia National Laboratories [6,7]. Implementation of the 
digital NIR camera system has resulted in NIR imagery with improved dynamic range, instantaneous field of view, and 
signal to noise [8-11]. The characterizations performed on the digital NIR system included radiometric, spatial, and 
spectral measurements using blackbody radiation sources and known targets [12,13]. An assessment of the collected 
data, using the digital NIR imaging sensor, for STS-1 19, STS-125, STS-128, STS-131, STS-132, STS-133, and STS-134 
missions, is presented. A majority of these missions contained a detailed test objective (DTO) to study boundary layer 
transition (BLT) [11]. The DTO BLT mission flew a modified tile with a protuberance to induce turbulent flow. This 
caused increased heating at the trip location and therefore required a high dynamic range NIR imaging system. 

The data were collected using a US Navy NP-3D Orion aircraft (STS-1 19, STS-125, STS-128, STS 132, and STS 133) 
and a land based Mobil Aerospace Reconnaissance System (MARS) (STS-131 and STS-134). The imaging assets are 
shown in Figure 2. A typical data collect contained approximately 10,000 to 13,000 image frames (20 - 27 gigabytes of 
data). Image quality is influenced by sun angle, aircraft vibration, Orbiter orientation, distance from Orbiter to imager, 
atmospheric attenuation, and tracking jitter. Nevertheless, excellent imagery is obtained, by selecting or “cherry 
picking” the best images. A histogram method is used to analyze the thermal imagery to determine the images of 
interest [12]. The histogram analysis method determines image saturation, frame dropouts and signal loss due to 
vignetting distortion. In addition, a technique to measure image sharpness using image eigenvalue decomposition 
approach is shown to determine image sharpness [14,15]. Selected images are then converted to quantitative 2- 
dimensional temperature images. The process used to convert the raw image counts to 2-dimensional temperature 
images is presented using atmospheric corrections [16,17]. Lastly, examples of HYTHIRM remotely measured 
temperatures are compared to the Orbiters’ onboard thermocouples. 



Figure 2. Imaging assets used for Orbiter re-entry data collects. 






2. SENSOR DESCRIPTION AND CHARACTERIZATION 


The digital NIR camera used in both Cast Glance and MARS imaging systems is the Prosilica GC1380H. The camera 
pixel array size is 1360x1024 with a frame rate of 30 Hz at full resolution through a GigE interface. The camera’s 
dynamic range is 12 bits. This wide dynamic range was valuable to minimize image saturation. The integration time can 
be varied from 10 jusec. to 60 sec. and is continuously variable in real time. The spectral response of the camera’s CCD 
sensor is 0.4 - 1.05 pm. A long pass filter with cut on at 0.850 pm is used to remove the spectral energy in the visible 
band. The acquired digital data file is stored in the flexible image transport system (FITS) format [18]. Frame 
acquisition time is stamped and the value is stored in the frame header. Custom data acquisition software was developed 
to control the camera’s integration time in steps down to 10 psec [12]. The characterizations performed on the digital 
NIR system included radiometric, spatial, and spectral measurements using blackbody radiation sources and known 
targets. These characterizations were used as input to the radiance model to predict the imaged NIR radiometric response 
of the Orbiter’s windward surface and allow for proper selection of the digital camera integration time to minimize pixel 
saturation. The radiometric characterization was performed using calibrated blackbody radiation sources set at various 
temperatures during low ambient light (night time or indoor) conditions. Radiometric calibration setup of the Cast 
Glance aircraft, typical blackbody setup orientation, and MARS calibration setup are shown in Figures 3a, 3b, and 3c 
respectively. The process involves the calibration of the measured radiance counts to actual temperature values. The 
calibration was performed in a temperature range of interest (up to 2400 °F) at specified sensor integration times. The 
results are calibration curves that can then be used to convert the measured pixel intensity to quantitative radiance 
values. The spatial characterization was performed during data collect conditions on selected stars. This technique 
allows for an estimation of the spatial response imaging system. The spatial point spread function (PSF) or blurring is 
modeled as a Gaussian function. Predetermined stars such as Sirius were observed to estimate the PSF. A spectrometer 
was used to determine the imaging system’s spectral response function. The spectral measurement was numerically 
combined with the cameras spectral response curve to determine the overall imaging system spectral response [13]. 



(b) 

Figure 3. Imaging assets during radiometric calibration. 


3. NIR IMAGERY ASSESSMENT 

After data collection an initial assessment of the data is made. This assessment of the quality of the data collected 
includes number of saturated pixels, dropped frames, integration time adjustments vs. image number, when acquisition 
of signal was acquired, and loss of signal. This analysis involves the entire sequence of images acquired around 10,000 
to 13,000 frames. Imagery of interest captured, such as reaction control system (RCS) firings, asymmetric boundary 
layer transition (ABET), DTO BET, contrail events, and best resolution images around the point of closest approach 
(PC A) are selected. A typical assessment is shown in Figure 4 for the HYTHIRM STS- 125 mission where PC A was 
around 36.5 nautical miles (NM) and Orbiter’s speed was Mach 15. As can be seen there are few saturated pixels. 
These images are a significant improvement over images shown in Figure 1. 

3.1 Image Histogram Analysis 

A more detailed assessment of the collected data is provided using a histogram analysis technique. The image histogram 
analysis method was used to analyze the NIR data frame by frame to determine the best resolution images by estimating 
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Figure 4. Imaging results for STS- 125 HYTHIRM data collect. 


the number of pixels on target. The number of pixels on target is determined by summing the pixels with counts above a 
threshold for each acquired frame. The equation is given as: 

4095 

N= Yfiincounts, (1) 

i — k 

where i is the counts, k is the counts threshold value, and N is the estimated Obriter’s pixel resolution. Because the sky 
background is cool, compared to the Orbiter’s windward surface which is hot (higher pixel counts), the number of pixels 
can be estimated by summing all the pixels associated with a value greater than approximately 3-5 times the average 
background counts. This is shown in Figure 5 where the peak resolution is compared to the slant range distance. The 
peak resolution image occurs at image # 6257 not at image # 6380 (image at minimum slant range distance). This 
indicates that at minimum slant range distance, the view angle to the windward surface is not optimal. 
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Figure 5. Imaging results for STS- 128 HYTHIRM data collect with Orb iter pixel resolution estimate. 














In addition, the image histogram analysis method was used to determine acquisition and loss of signal, RCS firings, and 
signal loss to vignetting distortion. Vignetting distortion occurs when the optical pointing system is looking up near the 
edge of a window. The optical system is located permanently on the aircraft’s starboard side. A combination of the STS 
132 ground track, banking angle during decent and location of the Sun during acquisition required the positioning of the 
Cast Glance aircraft to be on the non-optimal side of the Orbiter re-entry track. Although the aircraft bank angle and 
gimbaled pointing system can help to compensate for vignetting it could not be avoided for this mission. Vignetting 
causes a reduction in image brightness, in extreme cases spatial distortions, and ultimately loss of signal. This is shown 
in Figure 6. The image histogram analysis curve was used to determine signal loss due to vignetting and therefore 
provides a technique to avoid imagery with vignetting loss. 
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Figure 6. STS- 132 HYTHIRM data collect with pixel resolution estimate showing vignetting distortion. 


3.2 Image Sharpness Measurement using Eigenvalue Decomposition 

STS- 134 data collect provided never seen before imagery of the development and movement of a hypersonic boundary 
layer transition front at unprecedented high resolution (approximately 10.2 cm per pixel) [19]. This data was collected 
with the land based MARS imager. The combination of the Orbiter traveling at a high rate of speed along with 
atmospheric attenuation and tracking jitter can cause varying degrees of frame-to-frame blurring. The image histogram 
analysis is able to determine an optimal range, when the best resolution images are collected. However, it is not able to 
determine the amount of frame-to-frame blurring. The problem is that a sharper image would contain more bright pixel 
values thereby increasing the number of pixels above the threshold, however the collected light would be spread over 
fewer pixels. Similarly, a blurred image would increase the number of pixels above the threshold, because the light 
would be spread over more pixels, however the intensity would decrease thus lowering the total pixels. Image selection 
based on sharpness was done previously using a feature-based approach where the wing leading edges, nose, and body 
flap were used to determine sharpness [20,21]. Because STS- 134 data collect provided imagery at a much higher 
resolution, a eigenvalue value decomposition technique was used to determine sharpness. Image sharpness using 
singular values provides a statistical approach to measure sharpness [15]. This technique is applicable for Orbiter 
imagery because the blur function or PSF contributes to the low spatial frequencies and the details of interest are in the 
higher spatial frequencies. The singular value decomposition is given below in Eq. 2. 

A = U A V T (2) 

Where A is the image matrix which is real (image counts) and has dimension m x n (m > n), U contains the left singular 
vectors, A is the diagonal matrix containing the singular values of A arranged in decreasing order, and V contains the 









right singular vectors. The sharpness metric S is then given below in Eq. 3 [15]. 

k 

S = trace [ A t ] = X a , , ( 3 ) 

i=l 

Where an are the diagonal singular values and S is the sum of singular values up through k. The value k is determined by 
reconstructing the image matrix A where sufficient spatial information is restored for a representative sharp image. This 
is shown in Figure 7 for a sharp image collected during STS- 134 re-entry showing thermal gradients or “fingers” 
associated with turbulent flow on the port wing. These fine details allow for further study to determine the origin of 
theses gradients [19]. The image reconstruction is given in Eq. 4 where A k is the reconstructed image and u t and v t are 
the left and right singular column vectors 


A 



( 4 ) 


respectively corresponding to the singular values Because the diagonal matrix A contains the singular values in 
decreasing order most of the image information is contained up to k = 50 as shown in Figure 7 with the line plots 
showing most of the detail is recaptured. For each acquired image, the eigenvalues are determined and the sharpness S is 
calculated using Eq. 3. An example plot is shown in Figure 8 for the STS- 134 data. Figure 8 shows the frame-by-frame 
variation in the image sharpness due to the blurring as previously discussed. Approximately two-thirds or less of the 
collected images are typically not selected or considered for subsequent processing [20]. Another benefit of this 
technique is viewing a series of images. When observing a series of images, blurring can distort the observation of 
interest such as the formation and movement of a boundary layer front. By using the sharpness metric, the undistorted 
sharp images in the sequence of interest can be selected for review temporally. 
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Figure 7. STS- 134 imagery reconstructed to determine lvalue for sharpness metric with line plot comparison. 

3.3 Optimal Image Selection using Image Histogram Analysis and Image Sharpness Measurement 

Combining the image histogram analysis technique with the image sharpness metric allows for the selection of the 
optimal images in the data collected. The image histogram technique determines the range of images where the optimal 
resolution or most pixels on the windward surface of the Orbiter is observed. Within that range the sharpest image or 
images can then be determined quantitatively using the eigenvalue technique. The optimal images for STS- 119, STS- 
125, STS-128, STS-131, STS-132, STS-133, and STS-134 missions are presented in Table 1 and shown in Figure 9. 
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Figure 8. Plot of sharpness metric for STS- 134 data with example images. 
Table 1. Optimal image selection. 


Mission (Date) 

Imaging 

Asset 

Maximum Resolution 
Image Number Range 

Selected Sharp 
Image Number 

Distance 

(NM) 

Mach 

STS-119 (3/28/2009) 

Cast Glance 

3550 - 3750 

3562 

30.6 

8.5 

STS-125 (5/24/2009) 

Cast Glance 

6300 - 6500 

6391 

38.0 

14.3 

STS-128 (9/12/2009) 

Cast Glance 

6100-6400 

6127 

46.8 

14.8 

STS-131 (5/26/2009) 

MARS 

2622 - 2664 

2643 

67.4 

15.1 

STS-132 (5/26/2010) 

Cast Glance 

4900 - 5300 

4954 

44.9 

13.1 

STS-133 (3/09/2011) 

Cast Glance 

6900 - 7200 
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Figure 9. Selected optimal images for STS-119, STS-125, STS-128, STS-131, STS-132, STS-133, and STS-134. 











4. IMAGE PROCESSING 


4.1 Conversion to Temperature 

The process used to convert the raw image counts to quantitative temperature is shown schematically in Fig. 10. Image 
processing required selection of a series of frames for averaging (6 to 25 frames). Proper registration of multiple frames 
for averaging requires that the Orbiter orientation, aspect ratio, and optical blurring must be minimal within these 
frames. The selection of frames for averaging utilized a feature based approach as described by Spisz [20,21]. The 
results are similar to the sharpness metric discussed in this paper. A radiometric calibration curve is used to covert 
counts to a radiance image for a given integration time. The radiance is corrected for atmospheric path transmission, 
solar scattering from aerosols, and path radiance to produce the in-band radiance at the Orbiter’ s windward surface using 
MODTRAN atmospheric modeling. The object radiance was converted to temperature by using Planck’s black body 
radiation law and the estimated in-band averaged reaction cured glass (RCG) coated tile emissivity value of 0.65 [22]. 
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Figure 10. Conversion of image counts to temperature image. 


4.2 Comparison to Onboard Thermocouples 

Thermal imagery obtained from the HYTHIRM missions is compared to the Orb iters’ onboard thermocouples values for 
STS-132, STS-133, and STS-134 in Figures 11a, 12a and 13a respectively. The comparisons were performed on the 
thermocouples located along the centerline. The actual locations of the thermocouples are in 3 -dimensional space due to 
the curvature of the windward surface and using the centerline simplifies location of the thermocouples. The averaged 
magnitude difference error was 125°F for STS-132, STS-133, and STS-134 images and the thermocouple measurements 
were typically lower. These results are similar to previous errors obtained during STS-119, STS-125, and STS 128 
comparisons [12,20]. The thermocouple offset error is 20 °F cooler than the actual surface with uncertainties around +/- 
20°F [23], however there was significant scatter as shown in Figure lib. Over a region of laminar flow (temperatures 
should be uniformly decreasing spatially from nose to aft) the scatter in thermocouple measurements suggests some 
additional thermocouple offset errors that have not been accounted for. A more rigorous mapping technique allowing for 
3 -Dimensional thermography has been developed [24]. This will allow for the exact determination of the thermocouple 
locations, especially those away from the centerline. Lastly, as expected, the measurement of lower temperatures 
(around 1,200 °F and below) will produce greater relative errors mainly due to the reduced long wave quantum 
efficiency of the CCD detector. 




Figure 11. STS- 132 centerline temperature comparison to thermocouple data. 



Figure 12. STS- 133 centerline temperature comparison to thermocouple data. 



Figure 13. STS- 134 centerline temperature comparison to thermocouple data. 


5. CONCLUSIONS 

A method to analyze digital NIR imagery of a vehicle during hypersonic re-entry is presented. A histogram method is 
used to analyze the thermal imagery and is used to determine image saturation, frame dropouts and signal loss due to 
vignetting. The histogram analysis technique also determines the range of images during maximum pixel resolution. In 
addition, a technique to measure image sharpness using an eigenvalue decomposition approach is presented. The 
combination provided an automated method to select the optimal images. Lastly as shown in Figure 9 the image quality 
has improved greatly from the early mission data collects of STS-121 and STS-115 (Figure 1). The STS-134 data 
collect represents unprecedented spatial resolution that defines the future potential for remote quantitative thermography 
imaging for hypersonic research [25]. This will allow for development of future space vehicle designs. 
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